Add a `terminate` ingest processor #114157

PeteGillinElastic · 2024-10-04T17:08:01Z

This processor simply causes any remaining processors in the pipeline to be skipped. It will normally be executed conditionally using the if option. (If this pipeline is being called from another pipeline, the calling pipeline is not terminated.)

For example, this:

POST /_ingest/pipeline/_simulate
{
  "pipeline":
  {
    "description": "Appends just 'before' to the steps field if the number field
 is present, or both 'before' and 'after' if not",
    "processors": [
      {
        "append": {
          "field": "steps",
          "value": "before"
        }
      },
      {
        "terminate": {
          "if": "ctx.error != null"
        }
      },
      {
        "append": {
          "field": "steps",
          "value": "after"
        }
      }
    ]
  },
  "docs": [
    {
      "_index": "index",
      "_id": "doc1",
      "_source": {
        "name": "okay",
        "steps": []
      }
    },
    {
      "_index": "index",
      "_id": "doc2",
      "_source": {
        "name": "bad",
        "error": "oh no",
        "steps": []
      }
    }
  ]
}

returns something like this:

{
  "docs": [
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "doc1",
        "_source": {
          "name": "okay",
          "steps": [
            "before",
            "after"
          ]
        },
        "_ingest": {
          "timestamp": "2024-10-04T16:25:20.448881Z"
        }
      }
    },
    {
      "doc": {
        "_index": "index",
        "_version": "-3",
        "_id": "doc2",
        "_source": {
          "name": "bad",
          "error": "oh no",
          "steps": [
            "before"
          ]
        },
        "_ingest": {
          "timestamp": "2024-10-04T16:25:20.448932Z"
        }
      }
    }
  ]
}

closes #110218

This processor simply causes any remaining processors in the pipeline to be skipped. It will normally be executed conditionally using the `if` option. (If this pipeline is being called from another pipeline, the calling pipeline is *not* terminated.) For example, this: ``` POST /_ingest/pipeline/_simulate { "pipeline": { "description": "Appends just 'before' to the steps field if the number field is present, or both 'before' and 'after' if not", "processors": [ { "append": { "field": "steps", "value": "before" } }, { "terminate": { "if": "ctx.error != null" } }, { "append": { "field": "steps", "value": "after" } } ] }, "docs": [ { "_index": "index", "_id": "doc1", "_source": { "name": "okay", "steps": [] } }, { "_index": "index", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [] } } ] } ``` returns something like this: ``` { "docs": [ { "doc": { "_index": "index", "_version": "-3", "_id": "doc1", "_source": { "name": "okay", "steps": [ "before", "after" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448881Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [ "before" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448932Z" } } } ] } ```

github-actions · 2024-10-04T17:08:14Z

Documentation preview:

✨ Changed pages

elasticsearchmachine · 2024-10-04T17:21:51Z

Hi @PeteGillinElastic, I've created a changelog YAML for you.

elasticsearchmachine · 2024-10-04T17:24:45Z

Hi @PeteGillinElastic, I've updated the changelog YAML for you.

elasticsearchmachine · 2024-10-04T17:25:32Z

Hi @PeteGillinElastic, I've updated the changelog YAML for you.

elasticsearchmachine · 2024-10-04T19:19:12Z

Pinging @elastic/es-data-management (Team:Data Management)

PeteGillinElastic · 2024-10-07T08:24:09Z

@dakrone Are you the best person to review this, since we were talking about the reroute mechanism earlier?

dakrone

LGTM, thanks for adding this Pete!

elasticsearchmachine · 2024-10-08T16:41:04Z

💚 Backport successful

Status	Branch	Result
✅	8.x

This processor simply causes any remaining processors in the pipeline to be skipped. It will normally be executed conditionally using the `if` option. (If this pipeline is being called from another pipeline, the calling pipeline is *not* terminated.) For example, this: ``` POST /_ingest/pipeline/_simulate { "pipeline": { "description": "Appends just 'before' to the steps field if the number field is present, or both 'before' and 'after' if not", "processors": [ { "append": { "field": "steps", "value": "before" } }, { "terminate": { "if": "ctx.error != null" } }, { "append": { "field": "steps", "value": "after" } } ] }, "docs": [ { "_index": "index", "_id": "doc1", "_source": { "name": "okay", "steps": [] } }, { "_index": "index", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [] } } ] } ``` returns something like this: ``` { "docs": [ { "doc": { "_index": "index", "_version": "-3", "_id": "doc1", "_source": { "name": "okay", "steps": [ "before", "after" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448881Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [ "before" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448932Z" } } } ] } ```

A docs page for the `terminate` processor was added in elastic#114157, but the change to include it in the outer processor reference page was omitted. This change corrects that oversight.

A docs page for the `terminate` processor was added in #114157, but the change to include it in the outer processor reference page was omitted. This change corrects that oversight.

A docs page for the `terminate` processor was added in elastic#114157, but the change to include it in the outer processor reference page was omitted. This change corrects that oversight.

A docs page for the `terminate` processor was added in #114157, but the change to include it in the outer processor reference page was omitted. This change corrects that oversight.

This processor simply causes any remaining processors in the pipeline to be skipped. It will normally be executed conditionally using the `if` option. (If this pipeline is being called from another pipeline, the calling pipeline is *not* terminated.) For example, this: ``` POST /_ingest/pipeline/_simulate { "pipeline": { "description": "Appends just 'before' to the steps field if the number field is present, or both 'before' and 'after' if not", "processors": [ { "append": { "field": "steps", "value": "before" } }, { "terminate": { "if": "ctx.error != null" } }, { "append": { "field": "steps", "value": "after" } } ] }, "docs": [ { "_index": "index", "_id": "doc1", "_source": { "name": "okay", "steps": [] } }, { "_index": "index", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [] } } ] } ``` returns something like this: ``` { "docs": [ { "doc": { "_index": "index", "_version": "-3", "_id": "doc1", "_source": { "name": "okay", "steps": [ "before", "after" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448881Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [ "before" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448932Z" } } } ] } ```

A docs page for the `terminate` processor was added in elastic#114157, but the change to include it in the outer processor reference page was omitted. This change corrects that oversight.

This processor simply causes any remaining processors in the pipeline to be skipped. It will normally be executed conditionally using the `if` option. (If this pipeline is being called from another pipeline, the calling pipeline is *not* terminated.) For example, this: ``` POST /_ingest/pipeline/_simulate { "pipeline": { "description": "Appends just 'before' to the steps field if the number field is present, or both 'before' and 'after' if not", "processors": [ { "append": { "field": "steps", "value": "before" } }, { "terminate": { "if": "ctx.error != null" } }, { "append": { "field": "steps", "value": "after" } } ] }, "docs": [ { "_index": "index", "_id": "doc1", "_source": { "name": "okay", "steps": [] } }, { "_index": "index", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [] } } ] } ``` returns something like this: ``` { "docs": [ { "doc": { "_index": "index", "_version": "-3", "_id": "doc1", "_source": { "name": "okay", "steps": [ "before", "after" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448881Z" } } }, { "doc": { "_index": "index", "_version": "-3", "_id": "doc2", "_source": { "name": "bad", "error": "oh no", "steps": [ "before" ] }, "_ingest": { "timestamp": "2024-10-04T16:25:20.448932Z" } } } ] } ```

A docs page for the `terminate` processor was added in elastic#114157, but the change to include it in the outer processor reference page was omitted. This change corrects that oversight.

elasticsearchmachine added the v9.0.0 label Oct 4, 2024

PeteGillinElastic added :Data Management/Ingest Node Execution or management of Ingest Pipelines including GeoIP >feature auto-backport Automatically create backport pull requests when merged v8.16.0 labels Oct 4, 2024

Update docs/changelog/114157.yaml

4574150

PeteGillinElastic force-pushed the fix/110218 branch from df8c436 to 4574150 Compare October 4, 2024 17:30

PeteGillinElastic marked this pull request as ready for review October 4, 2024 19:18

elasticsearchmachine added the Team:Data Management Meta label for data/management team label Oct 4, 2024

PeteGillinElastic requested a review from dakrone October 7, 2024 08:23

dakrone approved these changes Oct 8, 2024

View reviewed changes

PeteGillinElastic merged commit 43e5258 into elastic:main Oct 8, 2024

PeteGillinElastic deleted the fix/110218 branch October 8, 2024 16:40

PeteGillinElastic mentioned this pull request Oct 8, 2024

[8.x] Add a terminate ingest processor (#114157) #114343

Merged

PeteGillinElastic mentioned this pull request Oct 9, 2024

Actually add terminate docs page #114440

Merged

PeteGillinElastic mentioned this pull request Oct 10, 2024

Add terminate ingest processor elastic/elasticsearch-specification#3002

Closed

mattc58 mentioned this pull request Oct 10, 2024

Create UI support for Terminate Processor elastic/kibana#195782

Closed

donoghuc mentioned this pull request Jul 7, 2025

Whitelist terminate processor in E2E tests. elastic/logstash-filter-elastic_integration#327

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add a `terminate` ingest processor #114157

Add a `terminate` ingest processor #114157

Uh oh!

PeteGillinElastic commented Oct 4, 2024 •

edited

Loading

Uh oh!

github-actions bot commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

PeteGillinElastic commented Oct 7, 2024

Uh oh!

dakrone left a comment

Uh oh!

elasticsearchmachine commented Oct 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add a terminate ingest processor #114157

Add a terminate ingest processor #114157

Uh oh!

Conversation

PeteGillinElastic commented Oct 4, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

elasticsearchmachine commented Oct 4, 2024

Uh oh!

PeteGillinElastic commented Oct 7, 2024

Uh oh!

dakrone left a comment

Choose a reason for hiding this comment

Uh oh!

elasticsearchmachine commented Oct 8, 2024

💚 Backport successful

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Add a `terminate` ingest processor #114157

Add a `terminate` ingest processor #114157

PeteGillinElastic commented Oct 4, 2024 •

edited

Loading